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Abstract 

Background: In recent years, there have been several studies, using a wide variety of methods, aimed at 
developing quality indicators for palliative care. In this Quality Indicators for Palliative Care study (Q-PAC study) we 
have applied a scientifically rigorous method to develop a comprehensive and valid quality indicator set which can 
contribute to a standardized method for use in other countries. 

Methods and design: Firstly, an extensive literature review identified existing international quality indicators and 
relevant themes for measuring quality in palliative care. Secondly, the most relevant of these were selected by an 
expert panel. Thirdly, those prioritized by the experts were scored by a second multidisciplinary expert panel for 
usability and relevance, in keeping with the RAND/UCLA-method, combining evidence with consensus among 
stakeholders. This panel included carers and policymakers as well as patients and next-of-kin. Fourthly, the draft set 
was tested and evaluated in practice for usability and feasibility; the indicators were then translated into 
questionnaires presented to patients, next-of-kin and care providers. To encourage the acceptance and use of the 
indicators, stakeholders, including national palliative care organizations, were involved throughout the whole 
project. 

Conclusion: Our indicator development trajectory resulted in a set of quality indicators applicable to all patients in 
all palliative care settings. The set includes patient and relative perspectives and includes outcome, process and 
structure indicators. Our method can contribute internationally to a more standardized and rigorous approach to 
developing quality indicators for palliative care. 

Keywords: Quality indicators, Quality measurement, Palliative care, Quality of care, End of life care, Hospice care, 
Outcome measures, Developing method 



Background 

Quality of care is of interest to everyone receiving or 
providing palliative care [1]. Evaluation of quality 
informs care providers, administrators and policymakers 
about whether patients and their families are receiving 
care that fits their needs [2-4], The challenge for quality 
improvement in palliative care is to develop effective 
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ways for the quick and efficient assessment of service 
performance and outcomes, as this facilitates the modifi- 
cation of services and practices in order to improve the 
quality of care at individual and institutional levels [5-8]. 
For this purpose, quality indicators specifically for pallia- 
tive care can be used to address issues unique to this 
type of care [9]. 

Although in the past decade a variety of studies have 
focused on quality indicators for palliative care, the 
methods found in the literature by which indicators were 
developed were not always clearly presented and not 
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always seemed to follow a systematic approach involving 
different steps of the development process [10-26]. Two 
recent systematic efforts have developed a nationwide 
quality register system to measure outcomes and quality 
of palliative care services: the Palliative Care Outcomes 
Collaboration (PCOC) -study in Australia [27], and a 
similar registration project in Sweden [28]. While these 
projects have resulted in very useful national quality 
monitoring systems, they do not strictly make use of 
quality indicators but mainly focus on outcome 
measures. Additionally they did not seem to aim for a 
level of comprehensiveness necessary to evaluate the 
various dimensions of palliative care. 

It appears that there is a need for a more standardized 
method of developing a comprehensive quality indicator 
set. Even at a more fundamental level, however, there is also 
a need for a common understanding of what quality 
indicators are. Quality indicators are well-defined and 
measurable aspects of care that give an indication of the 
quality of care delivered; they are generally expressed in a 
number or percentage, address a specific aspect of care or a 
related outcome, and are expressed at an aggregated level, 
often the level of care organisations [29-32]. They have to 
be clinically relevant, manageable and based on existing evi- 
dence, or consensus in the absence of such evidence [30]. 
Specific properties have to be described: a numerator (ie for 
which patients the indicator is positive) and a denominator 
(the group of patients being evaluated), a threshold value as 
a performance standard and exclusion criteria. Validated 
measurement instruments and relevant outcome measures 
are preferably used [33]. Confusion persists about outcome 
measures, measurement instruments and quality indicators, 
with these concepts often used interchangeably and mis- 
takenly. Outcome measures are an essential component 
of quality, providing a way to evaluate patient- or family- 
level status and responses to treatment, measured on an 
individual level [34]. However, as quality indicators, they 
do not have standardized specifications detailing the 
eligible population, data collection procedures and types of 
analysis needed to calculate them [24] (see Table 1). 

Following the model proposed by Donabedian, quality 
indicators should be monitored in all three major areas of 
health care: structure, process and outcome [35]. Struc- 
ture refers to the organization of care (eg presence of a 
written policy about visiting patients in the ICU), whereas 
process refers to the interaction between providers and 
receivers of care (eg psychosocial support being offered 
within the first 72 hours of admission to the ICU) [36]. 
Outcomes of care are at the level of the beneficiary of that 
care (eg adequate treatment of pain) [36]. 

With good examples of systematic and scientifically 
rigorous development trajectories for quality indicators 
for palliative care lacking in the literature, we intend this 
article to contribute to the establishment of a scientifically 



rigorous standardized and applicable method of measur- 
ing usable quality indicators. We do this by describing the 
protocol of a study developing a quality indicators set for 
palliative care for adults in Flanders (the Flemish speaking 
part of Belgium), applicable in all settings providing pallia- 
tive care, thus providing an example usable by other 
countries interested in monitoring the quality of pal- 
liative care. 

Methods and design 

In order to develop a comprehensive set of quality 
indicators to monitor the quality of palliative care in 
Flanders, four phases were followed in the development 
process: 1) identification of existing quality indicators, 2) 
development of a framework for quality of palliative 
care, 3) indicator selection by expert consultations and 
4) testing the draft quality indicator set in palliative 
practice (Figure 1). The protocol of the present study 
was approved by the Ethical Review Board of Brussels 
University Hospital of the Vrije Universiteit Brussel. 

Phase 1: identification of existing quality indicators 

To get an overview of previous attempts to develop qual- 
ity indicators for palliative care, a thorough systematic lit- 
erature review was performed, building on a review done 
in 2007 by Pasman et al [36]. With the field of quality 
indicators and the literature on quality measurements in 
palliative care developing quickly, it was important to up- 
date the existing review using the same method and 
search strategies but including the most recent literature 
published between December 2007 and May 2009. 

As in the original review, we searched four inter- 
national databases (Pubmed, PsychlNFO, EMBASE and 
CINAHL). Two independent reviewers (KL and JC) 
followed the same procedures as in the original review 
by Pasman et al (results of this review update are 
integrated into a paper published in Journal of Pain and 
Symptom Management [37]). 

Phase 2: identifying domains and themes of quality 
palliative care 

To achieve eventual comprehensiveness in the set of 
quality indicators we conceived a conceptual framework 
based on publications focussing on the determination of 
high quality palliative care consisting of several domains 
[38-40]: 1) physical, 2) psychological, social and existen- 
tial, 3) information, communication, planning and deci- 
sion making with patients, 4) with family and 5) with 
other carers, 6) type of care, 7) coordination and con- 
tinuity, 8) support of friend or family carers and 9) 
structure of care (Table 2). Within these nine domains, 
we distinguished several themes (see Table 3 for an ex- 
ample of one domain with different themes). 
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Table 1 Definitions and examples of outcome measures, quality indicators and quality measurement instruments 



Outcome measure/ variable 



Quality indicator 



Quality measurement instrument 



Definition 



An essential component of quality 
whereby the focus lies on patient's or 
relatives outcomes, measured at an 
individual level. 



Characteristics Concerns outcomes of care 



Example 



In palliative care, outcome measures 
provide a way to evaluate patient- and 
family-level status and response to 
treatment for symptoms and conditions in 
physical, psychological and other domains 



Individual level 
Pain intensity 

Quality of life Comfort 

Patient's appraisal of the quality of care 
Relative's appraisal of the quality of care 



Well defined and measurable aspect of 
care, generally expressed in a number or 
percentage, addressing a specific aspect of 
care or a related outcome. Quality 
indicators are usually expressed on 
aggregated level. 

Concerns structure, process or outcomes 
of care 

Clinically relevant, manageable and based 
on existing evidence, or if not applicable 
on consensus 

Contains standardized specifications 
detailing the eligible population, data 
collection procedures and types of 
analyses needed to calculate the indicator 

Aggregated level 

Percentage of patients with moderate to 
severe pain [10] 

Extent to which patients indicate that 
caregivers respect their life stance [10] 

Extent to which direct relatives felt 
supported by the caregivers immediately 
after the patient's death [10] 



Instruments that can be used to monitor 
quality of care. 



Concerns structure, process or outcomes 
of care 

These instruments are used to measure 
outcome measures on individual level as 
well as quality indicators on aggregated 
level Individual and aggregated level 



Numerated rating scales (NRS) 

McGill Quality of Life Questionnaire [44] 

VOICES questionnaire [42,43] 



In order to select the themes most important to quality 
palliative care we held a consultation of experts representing 
all relevant actors in palliative care, including patient and 
family perspectives. The panel consisted of nurses and other 
caregivers from all different specialized palliative care 
services, physicians from different specialities, coordinators 
of the Flemish Palliative Care networks, volunteers, spiritual 
and religous counsellors, bereaved family members and 
representatives of palliative patients i.e. large patient 
organisations such as the Flemish Cancer League and the 
Flemish Advisory Board for the Elderly. 

The panel was asked to score each theme, as identified 
through the literature, on a scale from 1 (least important) 



to 5 (most important). Additionally we asked them to se- 
lect the three most important themes per domain 
(questionnaires available upon request). Per domain 
themes were ranked from most important to least import- 
ant based on the mean scores and the frequency with 
which the theme was indicated as most important within 
the domain. Based on the mean scoring (highest tertile 
across all domains) a number of themes were selected. To 
be selected, there had to be consensus among the panel 
members for each indicator, i.e. not more than four 
members scoring outside the mean range [1-9]. With 
every domain not then represented, we included one add- 
itional theme most often indicated as one of the three 



Phase 1 : identification of 
existing quality indicators(by I 
systematic review) 



Phase 2a: developing a 
framework with important 
dimensions and themes 



[ Indicators from literature 
i put in selected themes 
T 



Phase 3: experts' selection of 
indicators (by RAND method) 



Phase 2b: literature search and 
experts' selection of themes 
(within framework) 



.a 



Phase 4: developing a draft quality indicator set and testing it in practice 
Figure 1 Flowchart of the 4 phases of the Flanders Study to develop Quality Indicators for Palliative Care. 
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Table 2 Selected domains for the quality of palliative 
care in Flanders 

Domain 

1. Physical treatment and care 

2. Psychological, social and existential treatment and care 

3. Information, communication, planning and decision making with the 
patient 

4. Information, communication, planning and decision making with the 
family 

5. Information, communication, planning and decision making with 
other caregivers 

6. Type of care at the end of life 

7. Coordination and continuity of care 

8. Support of family and informal caregivers 

9. Structure of care 



Table 3 Example of themes within domains: Domain 1 
'Physical treatment and care' 

Process of care 

1. To measure or evaluate general symptom burden 

2. To measure or evaluate pain 

3. To measure or evaluate nausea and problems with digestion 

4. To measure or evaluate fatigue and insomnia 

5. To measure or evaluate decubitus 

6. To measure or evaluate appetite 

7. To measure or evaluate problems with respiration/tightness of the 
chest 

8. To measure or evaluate delirium 

9. To measure or evaluate complaints on the mouth 

10. To treat or care for physical symptoms once observed with the 
patient 

1 1 . Mentioning in the patient's file the presence of symptoms 

12. Mentioning in the patient's file the offered treatment and/or 
medication for the purpose of physical problems 

13. Mentioning in the patient's file the result of the offered treatment 
and/or medication for the purpose of physical problems 

Outcome of care 

1. Low general symptom burden 

2. Absence of pain 

3. Absence of problems with respiration/ tightness of the chest 

4. Absence of delirium 

5. Absence of decubitus 

6. Absence of fatigue and insomnia 

7. Absence of complaints of the mouth 

8. Absence of nausea and problems with digestion 

9. A good appetite 



most important themes within the domains that were 
underrepresented. We choose to preserve the nine 
domains because the research team decided that all 
domains should be covered in order to have a comprehen- 
sive set. Additionally, experts were able to identify add- 
itional themes that did not come out of the literature 
review in a separate box for each domain and these were 
added if consensus among the experts was reached (i.e. 
more than one expert added this theme). 

Phase 3: indicator selection by expert consultation 

The selected themes were then translated into quality 
indicators (i.e. with standardized specifications detailing 
the eligible population, data collection procedures, nu- 
merator and denominator), using the quality indicators 
found in the systematic literature review in the first 
phase. Then a second consecutive multidisciplinary pal- 
liative care expert panel was organized for another selec- 
tion round. This panel, like the first panel, consisted of 
researchers in quality measurement or in palliative care, 
nurses and other caregivers in the field of palliative care, 
caregivers with a policy task in palliative care, bereaved 
family members and patient representatives. The experts 
could add quality indicators important to them not 
presented in the list and could discard those they found 
less important. For social and existential aspects of care, 
no indicators were found in the literature so experts 
were asked to suggest well-defined and measurable qual- 
ity indicators for further development. 

Unlike with the first expert consultation, we put to- 
gether the second panel following the appropriateness 
method of the Research ANd Development corporation 
in collaboration with the University of California at Los 
Angeles (RAND/UCLA) [41]. This is the only systematic 
method for selecting quality indicators combining evi- 
dence with consensus among stakeholders. We chose to 
follow this method because, particularly for palliative 
care, quality indicators have to be developed using other 
evidence alongside expert opinion [29] as this area of 
health care has a limited or methodologically weak evi- 
dence base. Within this RAND/UCLA-method we 
executed the five prescribed steps [42]: 

1. Comprehensive literature review on the topic (i.e. 
systematic literature review in phase 1) and 
recommendation of a preliminary set of quality 
indicators within the quality framework obtained 
through the first expert panel. 

2. Recruiting expert clinicians from professional 
organisations reflecting the variety of specialities in 
palliative care and inviting them to join a panel for a 
two-stage process to rate the validity of the 
indicators. These experts included a representative 
from a patient organisation and a next-of-kin (eg 
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partner, relative or friend most closely involved), as 
in the first panel round. 

3. Sending the draft indicators by post to the panel 
members, who rated the indicators in terms of 
appropriateness and relevance as measures of quality 
on a scale from 1 (not appropriate or not necessary) 
to 9 (very appropriate or very necessary) 
(questionnaires available upon request). 

4. Feeding back first-round scores to the panellists for a 
second round of scoring in a face-to-face panel 
meeting. Only indicators with a median score of 4, 5, 
6 or higher but not reaching consensus were 
discussed, modified where necessary and rescored. 
The indicators with a median score of 7, 8 or 9 were 
automatically added to the quality indicator set and 
modified where necessary by the researchers. 

5. Using the second-round scores to select only those 
indicators rated highly for validity and on which 
panel members had reached consensus. Indicators 
rated low or not reaching consensus among 
panellists were discarded. 

After this second face-to-face consultation a draft set 
of quality indicators was produced (Figure 1). 

Phase 4: Developing and testing the draft quality 
indicator set in palliative practice 

The next challenge was to test the draft indicator set in 
palliative care practice, in particular in order to assess its 
usability and feasibility (eg work load). Up until now, the 
draft set had been based on literature and expert opinion 
and hence needed operationalization and validation in 
practice, after which final adjustments could be made. 

The indicators were transferred into questionnaires for 
patients, medical staff and next-of-kin (eg partner, relative 
or friend most closely involved) (Table 4). In order to obtain 
all information needed to calculate the selected indicators 
we composed our questionnaires using previously validated 
questions from existing questionnaires when these existing 
questions covered the quality indicator well. We for in- 
stance used questions of the Views Of Informal Carers - 
Evaluation of Services (VOICES) Questionnaire [43,44] and 
the McGill Quality of Life Questionnaire [45], particularly 
as these questionnaires are frequently used and already 



validated for a large group of respondents. For those 
indicators where no validated questions were available 
we composed questions ourselves. In a pre-test among 
patients, family members and caregivers, we tested this not 
yet validated questions and made an estimation of the time 
needed to complete each questionnaire.When patients were 
not able to complete the questionnaire themselves, a proxy 
was assigned. 

The questionnaires were divided over the observa- 
tional units via four different palliative care services 
(representing all the different palliative care services in 
Flanders): the palliative home care team, the palliative care 
unit, the palliative support team in the hospital and the 
care home (only residents for whom a palliative trajectory 
was initiated). A selection of services was recruited via the 
existing palliative care networks in Flanders: a call was 
sent out by the coordinators of these networks and 
services could volunteer to participate. After recruitment, 
a contact person for each service was appointed to 
organize the mailing of the questionnaires together with a 
researcher from the research team. The intention was to 
capture a cross section of the quality of palliative care in 
the different services at a particular moment, so one inter- 
rogation only was conducted. This demanded that a ran- 
dom selection was made of patients receiving palliative 
care services as well of those who had received palliative 
care services and had died between six weeks and six 
months earlier; a representative snapshot of the quality of 
existing care was thus created, taking into account coinci- 
dental patient mix confounders. With the quality of care 
susceptible to change over time, such a snapshot forms a 
basic starting point. The choice of a single measurement 
also makes the measurement of quality less burdensome 
for staff. 

Four different types of questionnaire (available upon 
request), each measuring different quality indicators, 
were sent: 

- For adult patients currently receiving palliative care 

- For next-of-kin involved in the care of a patient who 
had died six weeks to six months previously 

- For the central formal care provider (a nurse in the 
palliative home care team and in the care home and 
a nurse or a physician in the palliative support team 



Table 4 Distribution between observational units and study units during the testing phase 

Observational unit 

Patient Proxy Next-of-kin Caregiver Care institution 

Study unit before death before death after death before death after death before death 

Care for the patient x x x 

Care for the next of kin x 

Process of care x x 

Structure of care x 
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and the palliative care unit) about the care of a 
patient currently receiving palliative care 
- For general practitioner of patients who had died in 
the previous six weeks to six months 

A maximum of 10 questionnaires was sent to any one 
nurse or physician. 

We were able to list which patients, care providers and 
next-of-kin had already participated by the identifier 
numbers on the returned questionnaires attributed to each 
selected patient at the time of inclusion. In collaboration 
with the coordinator of each service involved, reminders 
were sent: one to patients and next-of-kin and up to two 
to care providers. As the researchers have not disposed of 
the list with patient, caregiver and family member contact 
details, identifying individuals through the identifier 
numbers on the questionnaires was not possible. After the 
reminders were sent and the questionnaires returned the 
collected data was analysed and quality indicators were 
calculated for each participating service. 

The testing phase was then evaluated in terms of feasibil- 
ity and usability. First telephone interviews with non- 
responding family members and family physicians were 
held to assess reasons for non-participation. This was done 
by a third independent party in order to preserve anonym- 
ity of patients and family members from the researchers. 
The contact person of the palliative service communicated 
the phone numbers of family and physicians directly to this 
independent person by email, based on the identifier 
numbers received by the researchers. Furthermore 
interviews were conducted with all coordinators and con- 
tact persons to evaluate the usability of the indicators and 
the procedure. Before the start of the interview all contact 
persons were asked to question the other caregivers about 
their experiences with the testing and the length of the 
questionnaire (s) they completed. 

Based on the results of the test in terms of usability 
and feasibility, the indicator set, questionnaires and 
manual were adapted in order to be ready for use by the 
palliative services independently. We evaluated the in- 
strument combining qualitative interviews with the 
caregivers involved about the workload and future use of 
the set and of quantitative psychometric analyses of the 
dataset gathered (e.g. number of missing answers per in- 
dicator). Furthermore, on request of the field, the indica- 
tor set was divided into a required minimal data set with 
complementary modules, each set usable separately or 
combined by the palliative services interested in quality 
measurement and improvement. 

Discussion 

This article presents a method followed in Flanders (the 
Flemish speaking part of Belgium) for developing quality 
indicators for palliative care. The method comprises four 



phases: 1) a literature review identifying existing quality 
indicators, 2) a literature review and expert consultation 
to identify the important themes and domains for quality 
palliative care, 3) quality indicator selection by expert 
consultation following the internationally validated 
RAND method and 4) testing the draft quality indicator 
set in practice. This four-phase method combines the 
different phases from previous development processes in 
other countries (ie literature review, expert consultation, 
practical test) and additionally includes patient and next- 
of-kin perspectives, structure, process and outcome 
indicators and applicability to all patients in all palliative 
care services. 

The method developed here can contribute to the de- 
velopment of a standardized and functional set of quality 
indicators for palliative care in other countries as it 
meets the standards of scientific rigour and the level of 
comprehensiveness required. 

Choosing a multi phased method leading to 
comprehensiveness 

An important strength of the quality indicators develop- 
ment method described here is that it contains several 
phases to make the quality indicator set as comprehen- 
sive as possible, i.e. measuring quality of care for all 
adult palliative patients and across all domains relevant 
to palliative care. By combining a literature review on 
quality indicators with the building of a framework on 
important themes in palliative care, all aspects of pallia- 
tive care have been taken into account. Previous sets of 
quality indicators for palliative care have tended to target 
specific populations such as cancer patients or vulner- 
able older people [9,16,18,46-48]. Our set, like that 
developed in the Netherlands is characterized by applic- 
ability to all setting and to all adult patient groups re- 
ceiving palliative care [10] making it possible in the 
future, after assessing case mix adjusters, to compare 
scores of quality indicators not only over different 
disciplines but over different care settings. 

Choosing a combination of evidence through literature 
review and expert consultation 

As the thinking about quality indicators and the literature 
about quality of palliative care is developing quickly, it is 
necessary to review the field regularly, especially when be- 
ginning to develop a new set of quality indicators. In the 
literature we found researchers developing quality 
indicators who had not performed such a review but had 
started from an existing quality indicator set such as Saliba 
et al. [22] who had selected indicators feasible for nursing 
home quality assessment from the original ACOVE qual- 
ity indicator set. This raises the question of whether com- 
pletely new indicators need to be developed for a 
particular country or whether existing indicators can be 
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adjusted and validated in a set for that country [37]. 
Either way, a literature review can help to reveal existing 
indicators for further development and quality monitoring 
in any country. 

Consultation of experts in palliative care in order to 
select the most important elements and indicators for 
measuring quality is a key element in the methodology 
described here and scoring by experts is important in 
assessing the validity and usefulness of the proposed 
indicators. Additionally it has the advantage that con- 
sultation of relevant palliative care actors enhances the 
acceptance and face-validity of the resulting indicator set 
among the palliative care community [10]. The compos- 
ition of the panel should reflect all relevant actors within 
palliative care: those in medical practice and palliative 
care, policymakers, researchers, patient representatives 
and family. The latter two categories are often forgotten 
in quality indicator development but are essential to the 
creation of a comprehensive set of quality indicators 
reflecting all perspectives in palliative practice. The ex- 
pert consultation can best be done by the RAND/ 
UCLA-method [42]. 

Choosing inclusion of outcome, process and structure 
indicators and involvement from all perspectives 

Although measuring structure and process indicators 
from medical charts or administrative databases may be 
easier and cheaper, using only such sources for quality 
measurements may cause problems. Such databases can 
be limited, especially where indicators depend on precise 
documentation of issues such as communication, 
patient-reported outcomes or preferences [9]; such infor- 
mation can only be obtained from patients and their 
families themselves. On the other hand some researchers 
and care providers tend to see quality only through the 
perspective of the patient. This perspective does indeed 
provide the best indication of whether good quality care 
has been achieved [10], but data for calculating have to 
be derived from individual care users which, apart from 
the practical issues, also limits the possibility of measur- 
ing all relevant aspects of the quality of palliative care 
which might be susceptible to improvement. Therefore 
many authors [9,29,35,38,39,49] suggest that a quality in- 
dicator set, in order to be broad and comprehensive, 
must include structure and process as well as outcome 
indicators. We chose to translate our draft indicator set 
into questionnaires for the different respondent groups, 
patients, family and care providers, depending on which 
we considered would be the most reliable raters for 
selected quality indicators. Other quality researchers 
have based their evaluation of usability and feasibility on 
abstraction from patient records [11,14,23,25], thereby 
automatically limiting the comprehensiveness of an indi- 
cator set. 



Choosing a manageable testing of the quality indicator 
set 

A crucial stage in the development process is the testing 
and evaluation of the set for usability and feasibility (eg 
work load) in practice; before it can be taken up on a na- 
tionwide scale the set needs operationalization and valid- 
ation in palliative practice. Because of its extensive 
character, we felt it necessary to perform this testing phase 
in a small sample of palliative care service organisations. 

To stimulate acceptance and use of quality indicators, 
stakeholders from palliative practice should be involved 
as early as possible in the development process. In Flan- 
ders we also collaborated closely with the Flemish Feder- 
ation for Palliative Care, which helped to promote the 
project and enhance participation from care providers. 

Work in progress 

The quality indicator set for palliative care for adults in 
Flanders is now fully developed and ready to be used in 
practice to monitor quality of palliative care. Before 
scores of quality indicators can be validly compared 
across different services and care providers, however, 
significant case mix adjusters and the discriminative 
power of the indicators need to be assessed. Only if fu- 
ture research in a larger sample confirms the discrimina- 
tive power of the indicators can they be recommended 
for comparative information or benchmarking [10] while 
benchmarking itself can only be made possible by 
implementing the quality indicator set on a large scale 
in Flanders. 

A standardized method for the development of quality 
indicators for palliative care can also offer opportunities 
on the international level; providing internationally com- 
parable data [50]. 

Conclusion 

There is an increasing recognition of the need for quality 
indicators for palliative care in order to develop quality 
programmes across countries and to provide evidence 
to policymakers about the quality of the palliative care 
they are providing. However, a standardized method of 
developing quality indicators for palliative care has been 
lacking. In this paper we propose a method which meets 
the required level of scientific rigour and creates a sound 
basis for achieving the comprehensiveness needed in 
a set of quality indicators. The method suggested here 
combines standard literature analyses with multidiscip- 
linary expert consultations involving all relevant actors 
within palliative care, following the RAND /UCLA- 
method, the only validated systematic method for quality 
indicator selection. Those wanting to develop indicators 
in other countries can use this standardized method to 
further develop and validate quality indicators for pallia- 
tive care. 
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